<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=11"/>
<meta name="generator" content="Doxygen 1.9.4"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>Flow: flow::net_flow::Send_pacing_data Struct Reference</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr id="projectrow">
  <td id="projectalign">
   <div id="projectname">Flow<span id="projectnumber">&#160;2.0.0</span>
   </div>
   <div id="projectbrief">Flow project: Full implementation reference.</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.9.4 -->
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&amp;dn=expat.txt MIT */
var searchBox = new SearchBox("searchBox", "search",'Search','.html');
/* @license-end */
</script>
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&amp;dn=expat.txt MIT */
$(function() {
  initMenu('',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
/* @license-end */
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="namespaceflow.html">flow</a></li><li class="navelem"><a class="el" href="namespaceflow_1_1net__flow.html">net_flow</a></li><li class="navelem"><a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html">Send_pacing_data</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="summary">
<a href="#pub-types">Public Types</a> &#124;
<a href="#pub-methods">Public Member Functions</a> &#124;
<a href="#pub-attribs">Public Attributes</a> &#124;
<a href="structflow_1_1net__flow_1_1Send__pacing__data-members.html">List of all members</a>  </div>
  <div class="headertitle"><div class="title">flow::net_flow::Send_pacing_data Struct Reference</div></div>
</div><!--header-->
<div class="contents">

<p>The current outgoing packet pacing state, including queue of low-level packets to be sent, for a given <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">Peer_socket</a>.  
 <a href="structflow_1_1net__flow_1_1Send__pacing__data.html#details">More...</a></p>

<p><code>#include &lt;<a class="el" href="low__lvl__io_8hpp_source.html">low_lvl_io.hpp</a>&gt;</code></p>
<div class="dynheader">
Inheritance diagram for flow::net_flow::Send_pacing_data:</div>
<div class="dyncontent">
<div class="center"><iframe scrolling="no" frameborder="0" src="structflow_1_1net__flow_1_1Send__pacing__data__inherit__graph.svg" width="144" height="119"><p><b>This browser is not able to show SVG: try Firefox, Chrome, Safari, or Opera instead.</b></p></iframe>
</div>
<center><span class="legend">[<a href="graph_legend.html">legend</a>]</span></center></div>
<div class="dynheader">
Collaboration diagram for flow::net_flow::Send_pacing_data:</div>
<div class="dyncontent">
<div class="center"><iframe scrolling="no" frameborder="0" src="structflow_1_1net__flow_1_1Send__pacing__data__coll__graph.svg" width="626" height="303"><p><b>This browser is not able to show SVG: try Firefox, Chrome, Safari, or Opera instead.</b></p></iframe>
</div>
<center><span class="legend">[<a href="graph_legend.html">legend</a>]</span></center></div>
<table class="memberdecls">
<tr class="heading"><td colspan="2"><h2 class="groupheader"><a id="pub-types" name="pub-types"></a>
Public Types</h2></td></tr>
<tr class="memitem:ae7b8c15eaf55d2891bf160574fd41739"><td class="memItemLeft" align="right" valign="top">using&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#ae7b8c15eaf55d2891bf160574fd41739">Packet_q</a> = std::deque&lt; <a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html#aef7998db71c60eeb5d1e3d1a97c14886">Low_lvl_packet::Const_ptr</a> &gt;</td></tr>
<tr class="memdesc:ae7b8c15eaf55d2891bf160574fd41739"><td class="mdescLeft">&#160;</td><td class="mdescRight">Short-hand for FIFO of low-level packets. <code>queue&lt;&gt;</code> dumbly has no <code>clear()</code>, so just use <code>deque&lt;&gt;</code> directly.  <a href="structflow_1_1net__flow_1_1Send__pacing__data.html#ae7b8c15eaf55d2891bf160574fd41739">More...</a><br /></td></tr>
<tr class="separator:ae7b8c15eaf55d2891bf160574fd41739"><td class="memSeparator" colspan="2">&#160;</td></tr>
</table><table class="memberdecls">
<tr class="heading"><td colspan="2"><h2 class="groupheader"><a id="pub-methods" name="pub-methods"></a>
Public Member Functions</h2></td></tr>
<tr class="memitem:ab7f2dc947fc3c57a6a0e93efb33c8a08"><td class="memItemLeft" align="right" valign="top">&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#ab7f2dc947fc3c57a6a0e93efb33c8a08">Send_pacing_data</a> (<a class="el" href="namespaceflow_1_1util.html#ac3e89a8a271b0ddc76ac2a0ce488dea4">util::Task_engine</a> *task_engine)</td></tr>
<tr class="memdesc:ab7f2dc947fc3c57a6a0e93efb33c8a08"><td class="mdescLeft">&#160;</td><td class="mdescRight">Initializes data to initial state (no active time slice).  <a href="structflow_1_1net__flow_1_1Send__pacing__data.html#ab7f2dc947fc3c57a6a0e93efb33c8a08">More...</a><br /></td></tr>
<tr class="separator:ab7f2dc947fc3c57a6a0e93efb33c8a08"><td class="memSeparator" colspan="2">&#160;</td></tr>
</table><table class="memberdecls">
<tr class="heading"><td colspan="2"><h2 class="groupheader"><a id="pub-attribs" name="pub-attribs"></a>
Public Attributes</h2></td></tr>
<tr class="memitem:af1e8f11a36db3b967d9a5f61b115750a"><td class="memItemLeft" align="right" valign="top"><a class="el" href="namespaceflow.html#a9d9cc2eeb10d398cff5591d446b763b8">Fine_time_pt</a>&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a">m_slice_start</a></td></tr>
<tr class="memdesc:af1e8f11a36db3b967d9a5f61b115750a"><td class="mdescLeft">&#160;</td><td class="mdescRight">The time point at which the last pacing time slice began; or epoch if no packets sent so far (i.e., no time slices yet =&gt; undefined).  <a href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a">More...</a><br /></td></tr>
<tr class="separator:af1e8f11a36db3b967d9a5f61b115750a"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:af0c881081c6706dba393ef814386b143"><td class="memItemLeft" align="right" valign="top"><a class="el" href="namespaceflow.html#a48799f1263cdeedec125be51a3db2b79">Fine_duration</a>&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af0c881081c6706dba393ef814386b143">m_slice_period</a></td></tr>
<tr class="memdesc:af0c881081c6706dba393ef814386b143"><td class="mdescLeft">&#160;</td><td class="mdescRight">The length of the current pacing time slice period; this depends on congestion window and SRTT on the containing socket, at the time point <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a" title="The time point at which the last pacing time slice began; or epoch if no packets sent so far (i....">m_slice_start</a>.  <a href="structflow_1_1net__flow_1_1Send__pacing__data.html#af0c881081c6706dba393ef814386b143">More...</a><br /></td></tr>
<tr class="separator:af0c881081c6706dba393ef814386b143"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:a9cfd632911c3c4acd8e048dae28ded46"><td class="memItemLeft" align="right" valign="top">size_t&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#a9cfd632911c3c4acd8e048dae28ded46">m_bytes_allowed_this_slice</a></td></tr>
<tr class="memdesc:a9cfd632911c3c4acd8e048dae28ded46"><td class="mdescLeft">&#160;</td><td class="mdescRight">This many bytes worth of DATA packets may still be sent, at this time, within the time slice defined by <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a" title="The time point at which the last pacing time slice began; or epoch if no packets sent so far (i....">m_slice_start</a> and <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af0c881081c6706dba393ef814386b143" title="The length of the current pacing time slice period; this depends on congestion window and SRTT on the...">m_slice_period</a>; if this is less than the data size of <code>m_packet_q.front()</code>, then no more can be sent, and any packets that need to be sent must be sent in the next time slice (i.e., at time point <code>&gt;= m_slice_start + m_slice_period</code>).  <a href="structflow_1_1net__flow_1_1Send__pacing__data.html#a9cfd632911c3c4acd8e048dae28ded46">More...</a><br /></td></tr>
<tr class="separator:a9cfd632911c3c4acd8e048dae28ded46"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:aed68c9912e8c2ada5a9fcef0f9df2331"><td class="memItemLeft" align="right" valign="top"><a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#ae7b8c15eaf55d2891bf160574fd41739">Packet_q</a>&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#aed68c9912e8c2ada5a9fcef0f9df2331">m_packet_q</a></td></tr>
<tr class="memdesc:aed68c9912e8c2ada5a9fcef0f9df2331"><td class="mdescLeft">&#160;</td><td class="mdescRight">Queue of low-level packets to be sent to the remote endpoint, in order in which they are to be sent, with the head element the one to be sent next.  <a href="structflow_1_1net__flow_1_1Send__pacing__data.html#aed68c9912e8c2ada5a9fcef0f9df2331">More...</a><br /></td></tr>
<tr class="separator:aed68c9912e8c2ada5a9fcef0f9df2331"><td class="memSeparator" colspan="2">&#160;</td></tr>
<tr class="memitem:a100a406213a5edf2854440397719bb91"><td class="memItemLeft" align="right" valign="top"><a class="el" href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6">util::Timer</a>&#160;</td><td class="memItemRight" valign="bottom"><a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#a100a406213a5edf2854440397719bb91">m_slice_timer</a></td></tr>
<tr class="memdesc:a100a406213a5edf2854440397719bb91"><td class="mdescLeft">&#160;</td><td class="mdescRight">When running, <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#aed68c9912e8c2ada5a9fcef0f9df2331" title="Queue of low-level packets to be sent to the remote endpoint, in order in which they are to be sent,...">m_packet_q</a> is non-empty, <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#a9cfd632911c3c4acd8e048dae28ded46" title="This many bytes worth of DATA packets may still be sent, at this time, within the time slice defined ...">m_bytes_allowed_this_slice</a> &lt; data size of <code>m_packet_q.front()</code>, and the timer will fire when this time slice ends, and thus the head packet in <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#aed68c9912e8c2ada5a9fcef0f9df2331" title="Queue of low-level packets to be sent to the remote endpoint, in order in which they are to be sent,...">m_packet_q</a> must be sent.  <a href="structflow_1_1net__flow_1_1Send__pacing__data.html#a100a406213a5edf2854440397719bb91">More...</a><br /></td></tr>
<tr class="separator:a100a406213a5edf2854440397719bb91"><td class="memSeparator" colspan="2">&#160;</td></tr>
</table>
<a name="details" id="details"></a><h2 class="groupheader">Detailed Description</h2>
<div class="textblock"><p >The current outgoing packet pacing state, including queue of low-level packets to be sent, for a given <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">Peer_socket</a>. </p>
<p >This structure is a data store and not an object. It is a separate <code>struct</code>, instead of directly inside <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">Peer_socket</a>, to make the complex <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">Peer_socket</a> structure's definition shorter and easier to understand. Accordingly the <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> code dealing with pacing (and working on this structure and related data) is in a separate file (<a class="el" href="low__lvl__io_8cpp.html">low_lvl_io.cpp</a> instead of peer_socket.cpp or node.cpp).</p>
<h3>Background on packet pacing</h3>
<p >Empirically we've seen that sending out many UDP datagrams at the same time may cause extra &ndash; sometimes quite major &ndash; loss somewhere in the network. To combat that, we've seen simple packet pacing be effective (and I hear Linux kernel does it too). The idea is to keep datagrams being sent at below a certain dynamically adjusting rate. More specifically, if we want to (and, subject to the send window, are allowed to) send N equally sized available packets over T msec, it may prevent loss to send them equally spaced T/N msec apart, as opposed to sending a burst of N packets immediately. In other words the total send rate over time period T is the same in either case, but in the latter case this is accomplished smoothly instead of burstily. This is pacing: minimize burstiness without lowering overall throughput. The pending packets that cannot be sent immediately are added to a queue to be sent later.</p>
<p >Suppose we keep the pacing system as above; begin a pacing time slice of length T, and over that time period send packets spaced at T/N msec apart; once T is over, recompute T and N and repeat. Let T be decided. We shouldn't use N = [number of packets pending to send], since at any moment many more can come in and get queued (causing backup of the queue). The natural solution is to use T = round trip time (SRTT) and N = congestion window (CWND), since those by definition set the maximum rate we will allow packets to be sent out, because the network cannot support more. Thus, the interval between packets is S = SRTT / CWND.</p>
<p >To make this system more responsive, S = SRTT / CWND can be recomputed after queued packet is sent (instead of recomputing it every SRTT as mentioned above; as SRTT and CWND are continuously computed as acknowledgments come in, much more frequently than every SRTT).</p>
<p >More precisely, whenever we send a packet, start a timer for S = SRTT / CWND msec. While the timer is running any new packet available for sending should be added to the packet queue Q and not sent. Once tomer fires, send packet at top of Q, if any (and start the timer again, since packet was sent). If Q is empty, and the timer is not running, and a new packet is available for sending, it should be sent immediately. Finally, any packet available for sending until SRTT is known (i.e., before the first acknowledgment comes in) should be sent immediately (all packets after that are paced per above algorithm).</p>
<p >There is a technical limitation that complicates the above. The timer period S can easily be quite low, e.g., S = 100 msec / 100 = 1 msec. In the current implementation (boost.asio) there is no timer mechanism with that precision (platform-dependent, but see comment on <a class="el" href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6" title="boost.asio timer.">util::Timer</a> alias) (we can <em>measure</em> time with much better precision, but we cannot <em>schedule</em> actions). Therefore we cannot always schedule timer for S. Suppose the timer has a minimum scheduling precision of &lt; R. Since we mathematically cannot avoid burstiness if S = SRTT / CWND &lt; R, we must minimize it; namely, in that case, set S = R and, instead of 1 packet max during that period, allow N = R / (SRTT / CWND) packets during that period. Thus the simple "queue packet each time
it is available, if timer is running" policy changes to the slightly more complex "start counter
K = N when timer starts; send packet and decrement K each time packet is available, until K is
down to zero, then queue packet." Note that the latter reduces to the former when S &gt;= R (i.e., timer precision is not a problem).</p>
<p >Now to formally define the invariants guiding the algorithm: let [T, T + S) be a time slice, where T is the time point at which the first packet is sent once an SRTT is known. S = max(floor(SRTT / CWND), R), where SRTT and CWND (which is in units of max-block-sizes) are determined at time T. Each packet should be sent out as soon as possible after it becomes available for sending, subject to the following constraint: at most N = S / (SRTT / CWND) full-sized blocks' worth of packets may be sent during the time slice [T, T + S); any packets beyond that should be queued on packet queue Q (and sent based on the policy in his sentence, in order of Q). After the time slice is finished, the next time slice [T, T + S) is re-defined at the time T the first packet available for sending appears (at which time S and N are recomputed), and the above policy applies. This continues indefinitely.</p>
<h3>Packets other than DATA</h3>
<p >There is one final consideration. The above discussion implicitly treats all packets as DATA packets. Since SYN/SYN_ACK/SYN_ACK_ACK packets all occur before the first SRTT measurement is available (but see below to-do), they are not relevant. However, there are also ACK and RST packets. First, let's deal with RST; an RST is by definition the very last packet we can send on a connection. We have no desire for it to be dropped any more than a DATA packet; so it can be subject to pacing just like DATA and placed onto Q, etc. Since it's the last packet to be sent, there is nothing more to discuss. However, that would mean the sending of RST may occur after socket is in state CLOSED. While we could make that work, it introduces unnecessary complication; we prefer to consider the <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">Peer_socket</a> quite dead once it is in CLOSED. Therefore I choose to not pace RSTs: send it immediately. This MAY put the RST ahead of other queued packets (ACK, DATA), but since RST indicates a sudden connection break, this is more or less acceptable (and those other packets will not be sent, as we stop sending paced packets once in CLOSED). It may also increase the chance the RST is dropped, but I highly doubt it's significant in reality.</p>
<p >What about ACKs? Probably we must place them onto Q, since sending them earlier would at least mean changing the original order of <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a>'s intended outgoing packet stream. However, how many bytes, in comparison to (N * max-block-size), is an ACK worth? We could do this a couple of ways. It could be worth max-block-size, like a typical DATA packet; this is the conservative choice. However, it's probably not too realistic, since an ACK is much smaller than a full DATA. It could be worth 0 bytes (i.e., send any ACK as soon as it's at the head of Q, regardless of time slice), which is closer to the truth but not true (the other way). Or it could be worth ~K bytes, where K is the approximate size of its payload (not counting the common header data). That is most realistic, but would throw off a bit the appealing exact math when all DATA packets are full-sized. What to choose? The answer is probably not too important, because in most applications there will be no significant mixing of outgoing DATA and ACK packets (since either one side or the other side is usually sending, not both simultaneously). I choose to go with ACK = 0 bytes, since it is fairly close to realistic yet also simple to implement.</p>
<h3>Thread safety</h3>
<p >Meant to be accessed from thread W only.</p>
<dl class="todo"><dt><b><a class="el" href="todo.html#_todo000061">Todo:</a></b></dt><dd>One way to provide finer pacing, in the situation where we must send more than 1 packet within the minimal schedulable timer period, is to rely on events other than timer clock ticks that occur more frequently. Namely, ACKs will possibly arrive faster that the minimal timer schedulable ticks do. Therefore, when a time slice begins, we can send no packets (instead of all the packets that must be sent in that time slice, as is the case currently). Then over the course of the time slice we can opportunistically &ndash; upon each ACK send the proper number of queued packets up to that point into the time slice (this can be computed accurately, because we can MEASURE time very accurately &ndash; just can't schedule things as accurately). Finally, once the time slice does expire, send off any packets still remaining for that slice. This way we can opportunistically provide finer pacing, where the ACK clock allows it.</dd></dl>
<dl class="todo"><dt><b><a class="el" href="todo.html#_todo000062">Todo:</a></b></dt><dd>Obtain the first RTT measurement based on SYN-SYN_ACK or SYN_ACK-SYN_ACK_ACK interval; then SRTT will be available from the start, and packet pacing can be enabled with the very first DATA packet, avoiding the initial (albeit small) burst of DATA packets. This is pretty easy.</dd></dl>
<dl class="todo"><dt><b><a class="el" href="todo.html#_todo000063">Todo:</a></b></dt><dd>As noted above we perform packet pacing but currently choose to assign a value of 0 bytes to an ACK. That is, while we do preserve the order of DATA and ACK packets &ndash; if both happen to be in the outgoing stream &ndash; we do not delay the sending of the ACK once it is the next packet to be sent out. However, even so, an ACK's sending may be delayed by the pacing applied to DATA packets intermixed with it. Therefore the ACK delay measurement we take here may be incorrect (too low) in that case. This can cause overestimated RTTs on the sender's side. The to-do is to correct the ACK delay value in a given ACK by adding the pacing delay (if any) of the ACK to the individual ACK delays within it. Conceptually this is similar to the <code>m_sent_when</code> value being set when choosing to send a DATA packet and then corrected in the pacing module later. This to-do is not important until we in practice start mixing sending and receiving at the application layer... but still &ndash; it's worth knowing that there is a design bug here.</dd></dl>
<dl class="todo"><dt><b><a class="el" href="todo.html#_todo000064">Todo:</a></b></dt><dd>My quick look into Google BBR, the recent advanced congestion control algorithm, suggests that the pacing algorithm can be used as part of congestion control rather than something merely applied after congestion control has made the CWND decision. I believe the idea would be this: If we can somehow very reliably determine the available bandwidth, which Google BBR does purport to do, then instead of controlling send rate via CWND, instead one could apply it to this pacing module. It is a slight modification of the algorithm described above: instead of the rate of sending being equal to CWND/SRTT, make it equal to the bandwidth determined through Google BBR. If I understand correctly, Google BBR does maintain a CWND as well, but this is a safety upper bound, in case the computed bandwidth is too high to be practical (in this case CWND will slow down data being delivered to the pacing module &ndash; but, normally, this would not be the case, and the pacing module would be, in practice, the thing controlling the sending rate). Note this is a pretty big conceptual change; in particular, it would make the queue Q grow much larger than what we see currently.</dd></dl>
<h3><code>Timer</code> vs <a class="el" href="namespaceflow_1_1util.html#a74d4a66fb00055be4e1bd6ee98e04667" title="Schedule the given function to execute in a certain amount of time: A handy wrapper around Timer (asi...">util::schedule_task_from_now()</a></h3>
<p >In other places we have tended to replace a <code>Timer</code> with the far simpler <a class="el" href="namespaceflow_1_1util.html#a74d4a66fb00055be4e1bd6ee98e04667" title="Schedule the given function to execute in a certain amount of time: A handy wrapper around Timer (asi...">util::schedule_task_from_now()</a> (etc.) facility (which internally uses a <code>Timer</code> but hides its various annoyances and caveats). Why not here? Answer: These tasks are potentially scheduled <em>very</em> frequently; and the firing periods can push the limits of how small they can effectively be. Therefore, the perf caveats (as listed in <a class="el" href="namespaceflow_1_1util.html#a74d4a66fb00055be4e1bd6ee98e04667" title="Schedule the given function to execute in a certain amount of time: A handy wrapper around Timer (asi...">util::schedule_task_from_now()</a> doc header) apply in a big way &ndash; moreover, exactly when the effects would be felt the most (namely at high throughputs &ndash; which in turn is likeliest to peg process use). We should definitely not risk that in this case. </p>

<p class="definition">Definition at line <a class="el" href="low__lvl__io_8hpp_source.html#l00176">176</a> of file <a class="el" href="low__lvl__io_8hpp_source.html">low_lvl_io.hpp</a>.</p>
</div><h2 class="groupheader">Member Typedef Documentation</h2>
<a id="ae7b8c15eaf55d2891bf160574fd41739" name="ae7b8c15eaf55d2891bf160574fd41739"></a>
<h2 class="memtitle"><span class="permalink"><a href="#ae7b8c15eaf55d2891bf160574fd41739">&#9670;&nbsp;</a></span>Packet_q</h2>

<div class="memitem">
<div class="memproto">
      <table class="memname">
        <tr>
          <td class="memname">using <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#ae7b8c15eaf55d2891bf160574fd41739">flow::net_flow::Send_pacing_data::Packet_q</a> =  std::deque&lt;<a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html#aef7998db71c60eeb5d1e3d1a97c14886">Low_lvl_packet::Const_ptr</a>&gt;</td>
        </tr>
      </table>
</div><div class="memdoc">

<p>Short-hand for FIFO of low-level packets. <code>queue&lt;&gt;</code> dumbly has no <code>clear()</code>, so just use <code>deque&lt;&gt;</code> directly. </p>

<p class="definition">Definition at line <a class="el" href="low__lvl__io_8hpp_source.html#l00182">182</a> of file <a class="el" href="low__lvl__io_8hpp_source.html">low_lvl_io.hpp</a>.</p>

</div>
</div>
<h2 class="groupheader">Constructor &amp; Destructor Documentation</h2>
<a id="ab7f2dc947fc3c57a6a0e93efb33c8a08" name="ab7f2dc947fc3c57a6a0e93efb33c8a08"></a>
<h2 class="memtitle"><span class="permalink"><a href="#ab7f2dc947fc3c57a6a0e93efb33c8a08">&#9670;&nbsp;</a></span>Send_pacing_data()</h2>

<div class="memitem">
<div class="memproto">
<table class="mlabels">
  <tr>
  <td class="mlabels-left">
      <table class="memname">
        <tr>
          <td class="memname">flow::net_flow::Send_pacing_data::Send_pacing_data </td>
          <td>(</td>
          <td class="paramtype"><a class="el" href="namespaceflow_1_1util.html#ac3e89a8a271b0ddc76ac2a0ce488dea4">util::Task_engine</a> *&#160;</td>
          <td class="paramname"><em>task_engine</em></td><td>)</td>
          <td></td>
        </tr>
      </table>
  </td>
  <td class="mlabels-right">
<span class="mlabels"><span class="mlabel">explicit</span></span>  </td>
  </tr>
</table>
</div><div class="memdoc">

<p>Initializes data to initial state (no active time slice). </p>
<dl class="params"><dt>Parameters</dt><dd>
  <table class="params">
    <tr><td class="paramname">task_engine</td><td>IO service for the timer stored as data member. </td></tr>
  </table>
  </dd>
</dl>

<p class="definition">Definition at line <a class="el" href="low__lvl__io_8cpp_source.html#l01048">1048</a> of file <a class="el" href="low__lvl__io_8cpp_source.html">low_lvl_io.cpp</a>.</p>

</div>
</div>
<h2 class="groupheader">Member Data Documentation</h2>
<a id="a9cfd632911c3c4acd8e048dae28ded46" name="a9cfd632911c3c4acd8e048dae28ded46"></a>
<h2 class="memtitle"><span class="permalink"><a href="#a9cfd632911c3c4acd8e048dae28ded46">&#9670;&nbsp;</a></span>m_bytes_allowed_this_slice</h2>

<div class="memitem">
<div class="memproto">
      <table class="memname">
        <tr>
          <td class="memname">size_t flow::net_flow::Send_pacing_data::m_bytes_allowed_this_slice</td>
        </tr>
      </table>
</div><div class="memdoc">

<p>This many bytes worth of DATA packets may still be sent, at this time, within the time slice defined by <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a" title="The time point at which the last pacing time slice began; or epoch if no packets sent so far (i....">m_slice_start</a> and <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af0c881081c6706dba393ef814386b143" title="The length of the current pacing time slice period; this depends on congestion window and SRTT on the...">m_slice_period</a>; if this is less than the data size of <code>m_packet_q.front()</code>, then no more can be sent, and any packets that need to be sent must be sent in the next time slice (i.e., at time point <code>&gt;= m_slice_start + m_slice_period</code>). </p>
<p >Undefined (zero) if <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a" title="The time point at which the last pacing time slice began; or epoch if no packets sent so far (i....">m_slice_start</a> is undefined.</p>
<p >Why track it in bytes instead of multiples of max-block-size (as in the <code>struct</code> doc header)? Since the DATA packets in the queue may be of varying sizes, this allows us to more precisely (proportionally) count them against the <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#a9cfd632911c3c4acd8e048dae28ded46" title="This many bytes worth of DATA packets may still be sent, at this time, within the time slice defined ...">m_bytes_allowed_this_slice</a> budget. </p>

<p class="definition">Definition at line <a class="el" href="low__lvl__io_8hpp_source.html#l00212">212</a> of file <a class="el" href="low__lvl__io_8hpp_source.html">low_lvl_io.hpp</a>.</p>

<p class="reference">Referenced by <a class="el" href="low__lvl__io_8cpp_source.html#l00642">flow::net_flow::Node::sock_pacing_new_packet_ready()</a>, <a class="el" href="low__lvl__io_8cpp_source.html#l00757">flow::net_flow::Node::sock_pacing_new_time_slice()</a>, and <a class="el" href="low__lvl__io_8cpp_source.html#l00841">flow::net_flow::Node::sock_pacing_process_q()</a>.</p>

</div>
</div>
<a id="aed68c9912e8c2ada5a9fcef0f9df2331" name="aed68c9912e8c2ada5a9fcef0f9df2331"></a>
<h2 class="memtitle"><span class="permalink"><a href="#aed68c9912e8c2ada5a9fcef0f9df2331">&#9670;&nbsp;</a></span>m_packet_q</h2>

<div class="memitem">
<div class="memproto">
      <table class="memname">
        <tr>
          <td class="memname"><a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#ae7b8c15eaf55d2891bf160574fd41739">Packet_q</a> flow::net_flow::Send_pacing_data::m_packet_q</td>
        </tr>
      </table>
</div><div class="memdoc">

<p>Queue of low-level packets to be sent to the remote endpoint, in order in which they are to be sent, with the head element the one to be sent next. </p>
<p >Each new pending packet is to be added at the tail. Invariant: <code>m_packet_q.front()</code>, if it exists, is of type DATA after any <a class="el" href="classflow_1_1net__flow_1_1Node.html#a0556af278dc07a43f109fbd74ac64d5e" title="async_sock_low_lvl_packet_send_paced() pacing helper: Handles a DATA or ACK packet that was just pass...">Node::sock_pacing_new_packet_ready()</a> or <a class="el" href="classflow_1_1net__flow_1_1Node.html#a73cfaa8e5ed7fe845f759a30423a9cbf" title="async_sock_low_lvl_packet_send_paced() pacing helper: Given that we are currently in the pacing time ...">Node::sock_pacing_process_q()</a> call returns. (This is consistent with the aforementioned "do not re-order non-DATA packets, but do not pace them" philosophy.) </p>

<p class="definition">Definition at line <a class="el" href="low__lvl__io_8hpp_source.html#l00221">221</a> of file <a class="el" href="low__lvl__io_8hpp_source.html">low_lvl_io.hpp</a>.</p>

<p class="reference">Referenced by <a class="el" href="low__lvl__io_8cpp_source.html#l00642">flow::net_flow::Node::sock_pacing_new_packet_ready()</a>, and <a class="el" href="low__lvl__io_8cpp_source.html#l00841">flow::net_flow::Node::sock_pacing_process_q()</a>.</p>

</div>
</div>
<a id="af0c881081c6706dba393ef814386b143" name="af0c881081c6706dba393ef814386b143"></a>
<h2 class="memtitle"><span class="permalink"><a href="#af0c881081c6706dba393ef814386b143">&#9670;&nbsp;</a></span>m_slice_period</h2>

<div class="memitem">
<div class="memproto">
      <table class="memname">
        <tr>
          <td class="memname"><a class="el" href="namespaceflow.html#a48799f1263cdeedec125be51a3db2b79">Fine_duration</a> flow::net_flow::Send_pacing_data::m_slice_period</td>
        </tr>
      </table>
</div><div class="memdoc">

<p>The length of the current pacing time slice period; this depends on congestion window and SRTT on the containing socket, at the time point <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a" title="The time point at which the last pacing time slice began; or epoch if no packets sent so far (i....">m_slice_start</a>. </p>
<p >Undefined (<code>zero()</code>) if <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a" title="The time point at which the last pacing time slice began; or epoch if no packets sent so far (i....">m_slice_start</a> is undefined. </p>

<p class="definition">Definition at line <a class="el" href="low__lvl__io_8hpp_source.html#l00199">199</a> of file <a class="el" href="low__lvl__io_8hpp_source.html">low_lvl_io.hpp</a>.</p>

<p class="reference">Referenced by <a class="el" href="low__lvl__io_8cpp_source.html#l00642">flow::net_flow::Node::sock_pacing_new_packet_ready()</a>, <a class="el" href="low__lvl__io_8cpp_source.html#l00757">flow::net_flow::Node::sock_pacing_new_time_slice()</a>, and <a class="el" href="low__lvl__io_8cpp_source.html#l00841">flow::net_flow::Node::sock_pacing_process_q()</a>.</p>

</div>
</div>
<a id="af1e8f11a36db3b967d9a5f61b115750a" name="af1e8f11a36db3b967d9a5f61b115750a"></a>
<h2 class="memtitle"><span class="permalink"><a href="#af1e8f11a36db3b967d9a5f61b115750a">&#9670;&nbsp;</a></span>m_slice_start</h2>

<div class="memitem">
<div class="memproto">
      <table class="memname">
        <tr>
          <td class="memname"><a class="el" href="namespaceflow.html#a9d9cc2eeb10d398cff5591d446b763b8">Fine_time_pt</a> flow::net_flow::Send_pacing_data::m_slice_start</td>
        </tr>
      </table>
</div><div class="memdoc">

<p>The time point at which the last pacing time slice began; or epoch if no packets sent so far (i.e., no time slices yet =&gt; undefined). </p>
<p >The time period [<a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a" title="The time point at which the last pacing time slice began; or epoch if no packets sent so far (i....">m_slice_start</a>, <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a" title="The time point at which the last pacing time slice began; or epoch if no packets sent so far (i....">m_slice_start</a> + <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af0c881081c6706dba393ef814386b143" title="The length of the current pacing time slice period; this depends on congestion window and SRTT on the...">m_slice_period</a>) defines the range of time points in the time slice. </p>

<p class="definition">Definition at line <a class="el" href="low__lvl__io_8hpp_source.html#l00192">192</a> of file <a class="el" href="low__lvl__io_8hpp_source.html">low_lvl_io.hpp</a>.</p>

<p class="reference">Referenced by <a class="el" href="low__lvl__io_8cpp_source.html#l00642">flow::net_flow::Node::sock_pacing_new_packet_ready()</a>, <a class="el" href="low__lvl__io_8cpp_source.html#l00757">flow::net_flow::Node::sock_pacing_new_time_slice()</a>, and <a class="el" href="low__lvl__io_8cpp_source.html#l00841">flow::net_flow::Node::sock_pacing_process_q()</a>.</p>

</div>
</div>
<a id="a100a406213a5edf2854440397719bb91" name="a100a406213a5edf2854440397719bb91"></a>
<h2 class="memtitle"><span class="permalink"><a href="#a100a406213a5edf2854440397719bb91">&#9670;&nbsp;</a></span>m_slice_timer</h2>

<div class="memitem">
<div class="memproto">
      <table class="memname">
        <tr>
          <td class="memname"><a class="el" href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6">util::Timer</a> flow::net_flow::Send_pacing_data::m_slice_timer</td>
        </tr>
      </table>
</div><div class="memdoc">

<p>When running, <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#aed68c9912e8c2ada5a9fcef0f9df2331" title="Queue of low-level packets to be sent to the remote endpoint, in order in which they are to be sent,...">m_packet_q</a> is non-empty, <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#a9cfd632911c3c4acd8e048dae28ded46" title="This many bytes worth of DATA packets may still be sent, at this time, within the time slice defined ...">m_bytes_allowed_this_slice</a> &lt; data size of <code>m_packet_q.front()</code>, and the timer will fire when this time slice ends, and thus the head packet in <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#aed68c9912e8c2ada5a9fcef0f9df2331" title="Queue of low-level packets to be sent to the remote endpoint, in order in which they are to be sent,...">m_packet_q</a> must be sent. </p>
<p >In all other situations the timer is inactive. (Note that if the timer is not running, that does not mean any packet becoming available can be sent immediately; if the queue is empty and <code>m_bytes_allowed_this_slice == N</code>, the timer is off, but if a packet comes in with data size &gt; <code>N</code> before <code>m_slice_start + m_slice_period</code> [within the current time slice], then <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html#a100a406213a5edf2854440397719bb91" title="When running, m_packet_q is non-empty, m_bytes_allowed_this_slice &lt; data size of m_packet_q....">m_slice_timer</a> must be set to fire at time <code>m_slice_start + m_slice_period</code>.) </p>

<p class="definition">Definition at line <a class="el" href="low__lvl__io_8hpp_source.html#l00232">232</a> of file <a class="el" href="low__lvl__io_8hpp_source.html">low_lvl_io.hpp</a>.</p>

<p class="reference">Referenced by <a class="el" href="low__lvl__io_8cpp_source.html#l00841">flow::net_flow::Node::sock_pacing_process_q()</a>.</p>

</div>
</div>
<hr/>The documentation for this struct was generated from the following files:<ul>
<li>net_flow/detail/<a class="el" href="low__lvl__io_8hpp_source.html">low_lvl_io.hpp</a></li>
<li>net_flow/detail/<a class="el" href="low__lvl__io_8cpp_source.html">low_lvl_io.cpp</a></li>
</ul>
</div><!-- contents -->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Mar 28 2025 22:55:36 for Flow by&#160;<a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.9.4
</small></address>
</body>
</html>
